Skip to content

Allow ClawBench agents to read task files with a shell - #4

Merged
rgarcia merged 1 commit into
mainfrom
hypeship/allow-reading-task-files
Oct 13, 2026
Merged

rgarcia merged 1 commit into
mainfrom
hypeship/allow-reading-task-files

Conversation

@rgarcia

@rgarcia rgarcia commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

In the first full run of codex-kernel-mcp (129 tasks), 38 trials ended with the agent saying it couldn't read the task's credential files. 3 of those were still intercepted, and only 41 trials were intercepted in total.

The cause:

  • ClawBench's instruction says the task's files are under ./my-info/, but also says "Do NOT use command-line tools". Codex can only read files through its shell.
  • The previous harness in kernel-mcp-server resolved this conflict by telling the agent to use the credentials under ./my-info/. This repo dropped that line.
  • The kernel-mcp-local instruction then made it worse: it allowed reading "this one file" (the session-ID file), which implies other files are off limits.

Changes:

  • The ClawBench suite replaces its removed runtime block with one sentence: reading files with shell commands is allowed, and everything else goes through the browser. Reading ./my-info/ is a ClawBench concern, so the fix lives in the suite.
  • The toolset no longer limits file reading to the session-ID file.

Replaces #3, which closed when its base branch was deleted.

Validation

  • pytest (asserts the new sentence is in generated instructions), ruff, and bench prepare pass.
  • Full-arm rerun is linked in the comments.

@rgarcia
rgarcia merged commit d3cce1f into main Oct 13, 2026
2 checks passed
@rgarcia
rgarcia deleted the hypeship/allow-reading-task-files branch October 13, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant